Demo: Model Diagnostics (Self-Paced)

In this demonstration, you assess the assumptions for the previously fit stepwise selection model. In addition, you check for collinearity problems and for outliers and influential observations.

Reminder: If you restarted your SAS session, or it has timed out due to inactivity, you must re-create the course libraries, LOCALLIB and VST. To do this, run the AssignLibrary flow.

1. In SAS Viya, launch SAS Studio by clicking the Applications menu button (the three-by-three grid in the upper left corner), expand ANALYTICS LIFE CYCLE, and click Develop Code and Flows.
2. In the navigation panel on the far left, click the Explorer button (second from top). Expand Files >Home > workshop > VST. Double-click AssignLibrary.flw to open it.
3. Click the Run button on the flow toolbar.

Demo Steps

1. Return to the results of the Multiple_regression node in Correlation and regression.flw (last demonstration). Rerun the flow if necessary.
2. View the results on the Submitted Code and Results tab, focusing on the diagnostic tools and plots.




The second Parameter Estimates table has a column for variance inflation factors (VIF). These represent the strength of collinearity for each effect in the model. Collinearity is not a violation of the model assumptions, but it can result in model instability as well as other problems. A VIF >10 indicates a strong collinearity problem with the model.

PromCntCardAll has the highest VIF of 12.46, and GiftTimeFirst has the second highest VIF of 9.48. Collinearity inflates the variance of parameters relative to a model without collinearity, but for both of these variables, collinearity did not inflate the variances enough to make them nonsignificant. Both variables have p-values <0.0001. One approach to addressing collinearity problems is to remove variables involved in the collinearity, usually starting with the one with the highest VIF.





Predicted and observed values of the target match better than in the previous simple linear regression model. In the multiple regression model, smaller donations under $50 are not predicted well. They tend to be overestimated by the model with predictions generally over $50. In addition, higher predicted values tend to be overestimates of the observed donation amount.







The first column of the Fit Diagnostics plots can be used to assess whether the model assumptions have been met. The assumptions of independence, constant variance, and normality are necessary to be able to trust the p-values, standard errors, and confidence intervals from the model.

The residual by predicted value graph shows a non-random pattern. Predicted values over $175 all correspond to negative residuals. This suggests that the model form might be inadequate. Perhaps higher-order terms such as quadratic or interaction effects could improve the model fit. Residual variability might be getting smaller as the predicted values get larger. In addition, the residuals versus normal quantiles and the histogram of the residuals suggest deviations from a normal distribution. The histogram shows left skewness instead of a symmetric distribution. If statistical inference were the goal for this model, the researcher would likely modify this model to better meet the assumptions of regression.

The Cook's D and DFFITS plots show the presence of at least 1 to 3 influential observations. The three spikes that stand out in the Cook's D plot indicate cases that have a large influence on the entire set of parameter estimates (the intercept and the 15 slopes). The same three cases are also influential according to the DFFITS statistic. Large values of DFFITS that are well separated from other such values indicate cases that have a large influence on the predicted values.






The plots of residuals against each of the inputs tend to show the same pattern as residuals versus predicted values: many negative residuals at higher values of predictors and potentially nonconstant residual variance. These plots might indicate a misspecified model. Transformations of the inputs or adding quadratic terms can reduce the pattern of negative residuals at higher values of the predictors.







The last several scatter plots show two columns of dots. These correspond to dummy variables created for the two categorical predictors in the model.

Further effort could be used to improve the fit of this model, or other new models could be created and compared with the stepwise AIC model. These potential next steps are discussed later in this lesson.